Group communication — still complex after all these years

Group communication is used in many file systems and storage controller systems. Ideally, there could be a single group communication system that serves the needs of all these various projects—software that was built once, tested once, and deployed everywhere. In reality, each project builds its own custom group services component. There are three reasons for this: most systems require only a few group communication features (but not the same ones); the scalability requirements differ by several orders of magnitude between systems; and most storage systems have unique programming environment requirements. These differences in requirements imply that group communication is a toolkit that needs to be built differently each time to fit the particular target system requirements. To show to what extremes customization can go we take a look at two large-scale research storage systems: zFS and Palladio. These systems are interesting because they attempt to side-step the scalability boundaries of group communication by building mechanisms tailored to their systems, yet achieve strong semantics reminiscent of standard group communication systems.